New upstream fields by BrutuZ · Pull Request #906 · beeequeue/arm-server

BrutuZ · 2026-01-24T18:24:50Z

Fribb recently updated the lists generator, this PR aligns the output with the new output

This includes introducing a few new fields, ~~remapping thetvdb that was renamed to tvdb and thus failing to match~~ (removed on rebase after b47c3b6), as well as retiring notify-moe.

I couldn't get .alterTable() to work, hence opting to drop the entire table and recreate it on the DB migration. Didn't seem like much of a problem since it's always repopulated anyway

Before Changes (Live)

{
  "anidb": 5458,
  "anilist": 3390,
  "anime-planet": "amuri-in-star-ocean",
  "anisearch": 4445,
  "imdb": null,
  "kitsu": 2977,
  "livechart": 7110,
  "notify-moe": null,
  "themoviedb": 44298,
  "thetvdb": null,
  "myanimelist": 3390
}

{
  "anidb": 5459,
  "anilist": 3269,
  "anime-planet": "hack-g-u-trilogy",
  "anisearch": 4491,
  "imdb": "tt1164545",
  "kitsu": 2895,
  "livechart": 4721,
  "notify-moe": null,
  "themoviedb": 8864,
  "thetvdb": null,
  "myanimelist": 3269
}

After Changes

{
  "anidb": 5458,
  "anilist": 3390,
  "anime-planet": "amuri-in-star-ocean",
  "anisearch": 4445,
  "imdb": null,
  "kitsu": 2977,
  "livechart": 7110,
  "animenewsnetwork": 8720,
  "themoviedb": 44298,
  "themoviedb-season": 1,
  "thetvdb": 91021,
  "thetvdb-season": 1,
  "myanimelist": 3390,
  "simkl": 40868,
  "animecountdown": 40868,
  "media": "OVA"
}

{
  "anidb": 5459,
  "anilist": 3269,
  "anime-planet": "hack-g-u-trilogy",
  "anisearch": 4491,
  "imdb": "tt1164545",
  "kitsu": 2895,
  "livechart": 4721,
  "animenewsnetwork": 8719,
  "themoviedb": 8864,
  "themoviedb-season": null,
  "thetvdb": 79099,
  "thetvdb-season": null,
  "myanimelist": 3269,
  "simkl": 41283,
  "animecountdown": 41283,
  "media": "MOVIE"
}

BrutuZ · 2026-02-18T02:28:10Z

Rebased to address merge conflicts from b47c3b6
Ready for review @beeequeue

beeequeue

thanks for the effort, but this pr is too big to understand and merge at once. i would recommend splitting it into pieces

changing the source json file, explaining the differences between them
removal of the old fields
adding new fields
- if any of these are not 1-to-1 mappings they also need new endpoints to fetch them, like tvdb has

i can probably do it if you dont have time. :)

BrutuZ · 2026-02-18T18:56:27Z

The update to the documentation and tests really blew up the PR's line and file count 😅
The JSON has the same structure, just changed it due to the minor savings (a whopping ~50Kb 😆) of the minified / non pretty-print version. Not sure if it was previously available or a new introduction from Fribb's recent update to the generator.
The only non-unique fields (seasons and media type) introduced (AFAIK) would make little sense to have the reverse search since there's no relation, unlike the mentioned ones from anime-lists
I could also remove the DB migration on dev environment. Was useful while testing the schema updates

BrutuZ · 2026-02-19T02:08:44Z

Okay, I did split it in the 3 steps you described, but separate PRs means it will require your intervention because there will be merge conflicts and an extraneous and permanently empty notify-moe column in the database due to the separation of its removal and the new migration for new fields

BrutuZ · 2026-02-19T02:35:03Z

Force-pushed only the addition of new fields to this PR since I can only edit the target branch, not the origin.
Should you change your mind and decide the all-in-one approach is easier, I've kept that commit on https://github.com/BrutuZ/arm-server/tree/new-fields-full

beeequeue

thanks for keeping the pr more focused!
i have a lot of comments but they are mainly just to try and increase the code quality, dont take it too harshly. :^)

beeequeue · 2026-02-19T10:41:13Z

src/update.ts

 	thetvdb: handleBadValues(entry.tvdb_id),
+	"thetvdb-season": handleBadValues(entry.season?.tvdb),
+	simkl: handleBadValues(entry.simkl_id),
+	media: handleBadValues(entry.type),


im not sure if the media field is relevant as the project is for mapping IDs between services, and media type is something one would know from the source data
it could probably be skipped from the database

Two of the services (TVDB and TMDB) track both Movies and Series, and neither use unique IDs between the media types, meaning the same TMDB/TVDB ID can match a Series and a Movie, completely unrelated to each other.

beeequeue · 2026-02-19T10:42:56Z

src/update.ts

 export const removeDuplicates = (entries: Relation[]): Relation[] => {
-	const sources = (Object.values(Source) as SourceValue[]).filter(
-		(source) => source !== Source.TheTVDB && source !== Source.TheMovieDB && source !== Source.IMDB,
+	const sources = (Object.values(Source) as SourceValue[]).filter((source) =>


this should probably use .includes() or alternatively .has() together with changing NonUniqueFields to a Map for performance

beeequeue · 2026-02-19T10:43:39Z

src/update.ts

+			// eslint-disable-next-line unicorn/prefer-includes
+			if (NonUniqueFields.some((field) => field === source)) continue


same here, why disable the warning about using .includes()?

Both cases of Skill Issue™, .includes() gave a Type Warning and I couldn't figure out how to address it properly 😅

beeequeue · 2026-02-19T10:44:59Z

docs/openapi.yaml

            minimum: 0
            maximum: 50000000
            example: 1337
+      animenewsnetwork:


please make sure that the order of fields in relations (both in docs and code) are always alphabetical

I believe MAL was at the bottom in one place and it threw off my perception of anything else being alphabetized, I only noticed it yesterday with fresh eyes while splitting the PRs

beeequeue · 2026-02-19T10:45:37Z

src/db/db.ts

+	Source.TheMovieDBSeason,
+	Source.TheTVDB,
+	Source.TheTVDBSeason,
+	Source.Simkl,


is simkl unique or not?

I don't know for sure, no experience with the service but it does the whole "Global Movies and TV oh but also Anime" thing, so I preferred to err on the side of caution with non-unique

beeequeue · 2026-02-19T10:48:54Z

src/routes/v2/ids/handler.test.ts

 		myanimelist: id++,
+		simkl: id++,
+		animecountdown: id++,
+		media: `${id++}`,


media cant be a string number

I thought it was weird, but I followed notify.moe's model that is also alphanumerical and since the tests passed I didn't think about it

beeequeue · 2026-02-19T10:49:38Z

src/routes/v2/ids/handler.test.ts

+		animenewsnetwork: id++,
 		"notify-moe": `${id++}`,
 		themoviedb: id++,
+		"themoviedb-season": id++,


the seasons should probably be some number between 1-3 rather than an infinitely increasing number, you can hard code it for the tests

beeequeue · 2026-02-19T10:49:54Z

src/routes/v2/special/handler.test.ts

+		animenewsnetwork: id++,
 		"notify-moe": `${id++}`,
 		themoviedb: specialId ?? id++,
+		"themoviedb-season": specialId ?? id++,


same here, the seasons and media shouldnt be an id

beeequeue · 2026-02-19T10:50:23Z

src/routes/v2/include.ts

 			v.string(),
 			v.regex(/^[\-a-z,]+$/, "Invalid `include` query"),
 			v.minLength(1),
-			v.maxLength(100),


why is this change here?

With the added fields, the query length surpassed this limit.
IIRC it was a test with the include parameter and every field available

beeequeue · 2026-02-19T10:50:57Z

src/index.ts

 const runUpdateScript = async () => updateRelations()

-if (NODE_ENV === "production") {
+if (NODE_ENV === Environment.Prod) {


im not sure about this, it might be better to just remove the Environment constant altogether in another pr/commit

BrutuZ · 2026-02-19T16:21:34Z

I have a lot of comments but they are mainly just to try and increase the code quality, dont take it too harshly. :^)

Don't worry about it, I haven't dabbled with TS much so such comments and pointers are more than welcome 👍

BrutuZ mentioned this pull request Jan 27, 2026

Season object skipping Specials Fribb/anime-lists-generator#9

Open

BrutuZ changed the title ~~Update fields from Fribb's anime-lists~~ Fix TVDB and add new upstream fields Jan 31, 2026

BrutuZ force-pushed the new-fields branch from 67e3824 to c6cf5e8 Compare February 18, 2026 02:20

BrutuZ changed the title ~~Fix TVDB and add new upstream fields~~ Add new upstream fields Feb 18, 2026

BrutuZ force-pushed the new-fields branch from c6cf5e8 to fbaf4a4 Compare February 18, 2026 03:02

beeequeue requested changes Feb 18, 2026

View reviewed changes

BrutuZ changed the title ~~Add new upstream fields~~ New upstream fields and more Feb 19, 2026

This was referenced Feb 19, 2026

Switch to minified anime-list JSON #913

Open

Remove notify-moe from response #914

Open

Add New Fields

1b6ae95

BrutuZ force-pushed the new-fields branch from fbaf4a4 to 1b6ae95 Compare February 19, 2026 02:30

BrutuZ changed the title ~~New upstream fields and more~~ New upstream fields Feb 19, 2026

beeequeue requested changes Feb 19, 2026

View reviewed changes

fix migration, improve code structure in update.ts migration runner

0677ea7

		// eslint-disable-next-line unicorn/prefer-includes
		if (NonUniqueFields.some((field) => field === source)) continue

Uh oh!

Conversation

BrutuZ commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Changes (Live)

After Changes

Uh oh!

BrutuZ commented Feb 18, 2026

Uh oh!

beeequeue left a comment

Choose a reason for hiding this comment

Uh oh!

BrutuZ commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BrutuZ commented Feb 19, 2026

Uh oh!

BrutuZ commented Feb 19, 2026

Uh oh!

beeequeue left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BrutuZ commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

BrutuZ commented Jan 24, 2026 •

edited

Loading

BrutuZ commented Feb 18, 2026 •

edited

Loading

beeequeue left a comment •

edited

Loading